Skip to content

[AI] Add neural restore lighttable module for AI denoise and upscale#20523

Open
andriiryzhkov wants to merge 3 commits intodarktable-org:masterfrom
andriiryzhkov:neural-restore
Open

[AI] Add neural restore lighttable module for AI denoise and upscale#20523
andriiryzhkov wants to merge 3 commits intodarktable-org:masterfrom
andriiryzhkov:neural-restore

Conversation

@andriiryzhkov
Copy link
Contributor

This is a third and improved part of the original PR #20322.

Summary

  • Add a new lighttable/darkroom utility module (neural_restore) that provides AI-based image restoration using ONNX backend models
  • Supports two operations via a tabbed notebook UI: denoise (e.g. NIND UNet) and upscale (e.g. BSRGAN at 2x/4x)
  • Includes an interactive split before/after preview with draggable divider, on-demand generation (click to start), and a detail recovery slider that uses wavelet (DWT) decomposition to recover fine texture lost during denoising
  • Batch processing runs as a background job with progress reporting, cancellation support, tiled inference with overlap to avoid seam artifacts, and memory-aware tile size selection
  • Output is written as 32-bit float TIFF and automatically imported into the library, grouped with the source image
  • Register denoise-nind and upscale-bsrgan models in the AI model registry (ai_models.json)

Details

  • Built only when USE_AI=ON (cmake option, default OFF)
  • Preview thread uses atomic sequence counter for cancellation and is joined before cleanup to avoid use-after-free
  • Pixel pipeline: linear Rec.709 → sRGB before inference, sRGB → linear after; planar NCHW layout for ONNX models
  • Detail recovery: extracts luminance residual (original − denoised), applies per-band wavelet thresholding to separate noise from texture, blends filtered detail back at user-controlled strength
denoise upscale

Fixes: #19310

@esq4
Copy link
Contributor

esq4 commented Mar 14, 2026

Excellent! I was looking forward to it :)
But one question.
Output is written as 32-bit float TIFF and automatically imported into the library, grouped with the source image
That's very hard on the disk. Have you considered integrating it into the export like in CommReteris/nind-denoise?
With this implementation, the intermediate TIFF can be created in a temporary directory (I even placed it on tmpfs on my Linux) and deleted immediately after the export is complete.

@wpferguson
Copy link
Member

How do these models fit in with the "acceptable" model conversation?

If we merge a module that requires a model that's not "acceptable", then darktable is "endorsing"/requiring the non acceptable model.

@andriiryzhkov
Copy link
Contributor Author

@wpferguson:

How do these models fit in with the "acceptable" model conversation?

These models easily meet criteria for open-source AI. Please, see details here:

@andriiryzhkov
Copy link
Contributor Author

@esq4: Good point about disk usage. The current approach (import as grouped image) is intentional — it gives you a denoised/upscaled source that you can further develop in darkroom and compare via grouping. This is conceptually different from export, which is a final output step.

More flexibility would definitely be helpful here. What I think can be added as extra parameters:

  • Choice between 16-bit and 32-bit float TIFF (halves the size when full precision isn't needed)
  • Option to auto-import into the library or not
  • Maybe, configurable output directory (so you can point it to tmpfs or a fast scratch disk)

As for export-time denoising (like nind-denoise) — that's a different use case but a valid one. It could be added to the export module down the road as a complementary feature.

@wpferguson
Copy link
Member

BSRGAN only meets open source tooling and in the limitations it says Training datasets Flickr2K/WED/OST do not have explicit open-source licenses

If I look at some of the other models (SAM) they require data destruction in 3 months. Does that mean I have to destroy my edits? It also says no commercial use, so if I sell one of my images am I in violation?

I'm sorry, but this is a minefield. Somehow we need to decide a quick way to determine if a model is acceptable.

Do we use the OSAID, and if so what MOF? It seems Class I, Open Science, is fine. However Class II, Open Tooling, seems to come with lots of limitations/questions. If we decide to use Class II, how are we going to communicate the limitations to the users? Don't expect them to read, we already know how well that works.

@victoryforce
Copy link
Collaborator

If I look at some of the other models (SAM) they require data destruction in 3 months.

You are referring to clause 7 of SA-1B DATASET RESEARCH LICENSE. But that is a requirement for the training dataset, not the model. You can use that dataset for your research, for training your own model, but you can't keep it indefinitely. What's the problem?

Does that mean I have to destroy my edits?

Absolutely not. This is a conclusion from your statement above that is not true.

It also says no commercial use, so if I sell one of my images am I in violation?

Also no. The user is using the model and not the training dataset.

I'm sorry, but this is a minefield.

I'm also sorry, but this is NOT a minefield. Wanna see photos of what a minefield actually is? :)

Somehow we need to decide a quick way to determine if a model is acceptable.

Too vague. What is "acceptable"? Why should we decide what is acceptable for users and not the users themselves?

@KarlMagnusLarsson
Copy link

KarlMagnusLarsson commented Mar 15, 2026

Thank you for this PR. It works for me. The denoise and upscale functions work. I use the nind and bsrgan models from preferences -> AI after downloading them.

If I test this PR on top of git master branch I get fallback to CPU and can not activate NVIDIA CUDA. The denose and, even more so, upscale, are very heavy using CPU and I would say that a GPU is essential (not so for AI masks).

Workaround:
If I test this PR + PR #20522 (comment) (Enable GPU for ONNX runtime on GNU/Linux) then I can run NVIDIA CUDA and GPU. (Linux Debian, NVIDIA QUADRO RTX 4000 8GB, driver 550.163.01, CUDA 12.4).

I found a couple of preliminary issues:

  • The generated 32 bit floating point TIFF (deniosed, upscaled) do not contain a color profile. The normal darktable export module allows for picking color profile and and a rendering intent, EDIT: which is then stored in output file. I can se in details that we have: Pixel pipeline: linear Rec.709 → sRGB before inference, sRGB → linear after; planar NCHW layout for ONNX models. EDIT2: I can use the darktable export module and select linear Rec.709, if I want to (convert to) and embed linear rec 709 in output result file.
  • The EXIF data is not carried over from the source into the TIFF.

@piratenpanda
Copy link
Contributor

piratenpanda commented Mar 15, 2026

I'm getting:

2026-03-15 13:45:21.317621938 [W:onnxruntime:DarktableAI, migraphx_execution_provider.cc:1267 compile_program] Model Compile: Complete migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/concat.hpp:98: normalize_compute_shape: CONCAT: all input dimensions should match along axis 2 2026-03-15 13:45:21.328168818 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_8823259983746908714_2 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_8823259983746908714_2_2' Status Message: Failed to call function

when running this on my RX 6700 XT with HSA_OVERRIDE_GFX_VERSION=10.3.0

also bsrgan model won't download throwing an error about a missing integrity check as it couldn't download the chksum

@jenshannoschwalm
Copy link
Collaborator

Too vague. What is "acceptable"? Why should we decide what is acceptable for users and not the users themselves?

I guess you have recognized that there is some problem within the dt dev community about exactly this point. And yes, if we as devs decide - we don't want something for whatever reason that may be - it's not a user decision at all.

Maybe minefield was not the perfect wording. But you just should accept that for some of the long time devs the "who did and how a model was made" is absolutely critical. For me personally a razor-sharp yes-or-no.

@piratenpanda
Copy link
Contributor

piratenpanda commented Mar 15, 2026

CPU seems to work. But I think it should export to the default export folder. And will it in future also be possible to integrate it right into the normal export workflow?

Also without the override and AI set to off in the setting it crashes darktable:

darktable 
2026-03-15 14:09:51.924346434 [W:onnxruntime:DarktableAI, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
2026-03-15 14:09:51.968774532 [W:onnxruntime:, session_state.cc:1327 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.
2026-03-15 14:09:51.968788808 [W:onnxruntime:, session_state.cc:1329 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.
2026-03-15 14:09:52.013621785 [W:onnxruntime:DarktableAI, migraphx_execution_provider.cc:1262 compile_program] Model Compile: Begin

rocBLAS error: Cannot read /opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary.dat: Datei oder Verzeichnis nicht gefunden for GPU arch : gfx1031
 List of available TensileLibrary Files : 
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx908.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1101.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1150.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1100.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1201.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1030.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx90a.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx942.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx950.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1102.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1200.dat"
"/opt/rocm/lib/../lib/migraphx/lib/../../rocblas/library/TensileLibrary_lazy_gfx1151.dat"
Abgebrochen                (Speicherabzug geschrieben) darktable

I think it should just disable AI and make the checkbox grey or only allow CPU

@TurboGit
Copy link
Member

Note that as for the AI masking there is no models installed and downloaded automatically. All this stays in the hand of the end-users, so everyone can decide depending on its sensibility. I recognize that we have different perception about AI and I would never allow any model installed and/or delivered by default. Even the default build from source has no AI support, one need to pass a specific option to enable this.

I really think the current state is respecting everyone choice.

@andriiryzhkov
Copy link
Contributor Author

@piratenpanda: thank you for testing and feedback.

Regarding error you got. When running neural restore with MIGraphX provider on RX 6700 XT, the model compilation fails because MIGraphX can't handle dynamic shapes in concat operations with ORT_ENABLE_ALL optimization. I am thinking how we can optionally enable additional provider-dependent configs per model.

also bsrgan model won't download throwing an error about a missing integrity check as it couldn't download the chksum

That's interesting. It works on my side. Thinking of possible causes - can you double-check that in darktablerc you have correct repository config?

plugins/ai/repository=darktable-org/darktable-ai

Also without the override and AI set to off in the setting it crashes darktable:
...
I think it should just disable AI and make the checkbox grey or only allow CPU

Agree on disabling AI preferences when AI is disabled. This feature along with with proper AI actions block when AI is disabled implemented in PR #20534. It also should fix startup crash when AI is disabled.

I would really appreciate your help testing it further.

@wpferguson
Copy link
Member

I'm also sorry, but this is NOT a minefield. Wanna see photos of what a minefield actually is? :)

😢

Why should we decide what is acceptable for users and not the users themselves?

If the dev specifies a model necessary to run the module, it's not a user choice, it's a dev/darktable choice.

For example

I could take the GIMP lua script and replace gimp with photoshop and then add it to the lua-scripts repository and put the "blame" on the user for running it. The open source community would see that as darktable endorsing/requiring/encouraging the use of photoshop.

But...

Someone could build a script that lets you specify the external executable, such as ext_editor.lua, and you can decide for yourself what executables you want to run even if it's photoshop, capture one, dxo, etc, etc, etc. That is a user choice and darktable has no say in the user's decision.

The ONLY way the model can be a user choice is if darktable has NO SAY in the decision.

@piratenpanda
Copy link
Contributor

piratenpanda commented Mar 15, 2026

That's interesting. It works on my side. Thinking of possible causes - can you double-check that in darktablerc you have correct repository config?

Indeed it was the old one still. Changing to the right one it works fine

Agree on disabling AI preferences when AI is disabled. This feature along with with proper AI actions block when AI is disabled implemented in PR #20534. It also should fix startup crash when AI is disabled.

with the mentioned PR it does not crash anymore

@andriiryzhkov
Copy link
Contributor Author

The generated 32 bit floating point TIFF (deniosed, upscaled) do not contain a color profile.

The EXIF data is not carried over from the source into the TIFF.

Fixed - output TIFF now embeds linear Rec.709 ICC profile and source EXIF.

But I think it should export to the default export folder. And will it in future also be possible to integrate it right into the normal export workflow?

Added collapsible "output parameters" section: bit depth (8/16/32, default 16), catalog toggle, and output directory with darktable variable support (e.g. $(FILE_FOLDER)/darktable_exported). Core processing extracted to src/ai/restore.c as a reusable API - ready for export module integration.

@TurboGit
Copy link
Member

Windows CI failing with:

C:\Windows\system32\cmd.exe /C "cd . && D:\a\_temp\msys64\ucrt64\bin\cc.exe -Wall -Wno-format -Wshadow -Wtype-limits -Wvla -Wold-style-declaration -Wmaybe-uninitialized -Wno-unknown-pragmas -Wno-error=varargs -Wno-format-truncation -Wno-error=address-of-packed-member -fopenmp -march=native -msse2 -g -mfpmath=sse -O3 -DNDEBUG -O3 -ffast-math -fno-finite-math-only -fexpensive-optimizations  -shared  -Wl,--enable-runtime-pseudo-reloc -o lib\darktable\plugins\lighttable\libneural_restore.dll -Wl,--major-image-version,0,--minor-image-version,0 @CMakeFiles\neural_restore.rsp && C:\Windows\system32\cmd.exe /C "cd /D D:\a\darktable\darktable\build\lib\darktable\plugins\lighttable && D:\a\_temp\msys64\ucrt64\bin\cmake.exe -E make_directory .debug && objcopy --only-keep-debug D:/a/darktable/darktable/build/lib/darktable/plugins/lighttable/libneural_restore.dll D:/a/darktable/darktable/build/lib/darktable/plugins/lighttable/libneural_restore.dll.dbg && objcopy --strip-debug D:/a/darktable/darktable/build/lib/darktable/plugins/lighttable/libneural_restore.dll && objcopy --add-gnu-debuglink=libneural_restore.dll.dbg D:/a/darktable/darktable/build/lib/darktable/plugins/lighttable/libneural_restore.dll""
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0xba): undefined reference to `dt_ai_models_get_active_for_task'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x17b): undefined reference to `dt_ai_models_get_active_for_task'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x25b): undefined reference to `dt_ai_models_get_active_for_task'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x43c): undefined reference to `dt_ai_models_get_active_for_task'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x4bc): undefined reference to `dt_ai_models_get_active_for_task'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x55f): undefined reference to `dt_get_available_mem'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x1b96): undefined reference to `dt_alloc_aligned'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x1f5c): undefined reference to `dwt_denoise'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x228b): undefined reference to `dwt_denoise'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x235c): undefined reference to `dwt_denoise'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x23c8): undefined reference to `dt_alloc_aligned'
D:/a/_temp/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/15.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: bin/ai/libdarktable_ai.a(restore.c.obj):restore.c:(.text+0x26b1): undefined reference to `dwt_denoise'
collect2.exe: error: ld returned 1 exit status

@wpferguson
Copy link
Member

I really think the current state is respecting everyone choice.

Are we going to give the user any information about the model's licensing or are we putting that all on the user? Or are we expecting them to go digging to find out for themselves? If we don't at least point them at the information how are they supposed to make an informed choice.

Perhaps we include the OSAID rating and the MOF rating with a brief explanation (MOF I - conforms, MOF 2 - some exceptions, etc) and links?

@TurboGit
Copy link
Member

Perhaps we include the OSAID rating and the MOF rating with a brief explanation (MOF I - conforms, MOF 2 - some exceptions, etc) and links?

Probably a good idea, yes!

@TurboGit
Copy link
Member

But I think it should export to the default export folder.

So we are expecting this as the last edit step so done after all processing made to the image?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI Denoising

8 participants